Evaluating content placement options against benchmark placements

ABSTRACT

An online system evaluates the efficacy of one or more content placement slots within a content publisher in a publisher&#39;s page layout. To evaluate the efficacy of the content slot, the conversion rate at the slot and in a benchmark for selected content items published at the slot is identified to determine the ratio of conversions at the publisher slot and in the benchmark. For different content items, these conversion ratios may vary, and may further be weighed by a value contribution for each content item. Thus, to determine the value of the conversion ratios for a given publisher, the online system uses information about online system users and their interactions with third party content featured within different locations, or placements, within the publisher&#39;s page layout. The arrangement of slots in relation to one another can be manipulated by the publisher to yield optimal conversion rates for featured slots. For example, the publisher may reposition slots within a given page, or relocate slots to other pages, in order to increase conversion rates.

BACKGROUND

This invention relates generally to evaluating the effectiveness ofplacing content into content slots within a publisher, and particularlyto evaluating the efficacy of a publisher's content slot against abenchmark placement location.

Publishers provide content for users to consume in an interactivesystem. Such interactive systems may be websites, games, applications orother electronic interactive systems of interest to a user. Whenproviding the content to users, the publisher may develop a layout orother interactive flow in which the user interacts with the publisher.Within the layout, the publisher may provide one or more slots forcontent selected by another system or for placement of sponsoredcontent. For example, an application may include an interstitial slotfor placement of content between portions of the interactive publisher'scontent.

Publishers generally can select and control the location of thesecontent slots, which may impact a user's likely interaction with thecontent selected for and placed in the slot. Typically the user'sinteraction with the content placed in the slot is beneficial for thepublisher, for example to encourage a user to take a behavior related tothe content selected for the slot. The behavior may directly orindirectly benefit the publisher. For example sponsored content may beselected for the slot. Depending on where the slot is placed within thelayout of the interactive system, users of the interactive system may bemore or less likely to interact with the content. However, publisherstypically have lacked the tools to evaluate the efficacy of theseplacements, particularly because different content selected for the slotmay also have different interaction rates in the interactive systemrelative to other locations at which the selected content is shown (forexample, at other interactive systems). In addition, ineffective slotsselected by a publisher may often reflect positioning of a slot thatdetracts from or reduces user interaction with the publisher as a whole,reducing the quality of user experience with the publisher as a whole.

SUMMARY

An online system evaluates the efficacy of the content slots placedwithin a publisher's layout. The efficacy of placing content in aparticular slot can be evaluated relative to placements of the contentin the slot compared to the same placements in a benchmark system. Foreach content item to be placed in the slot, the content is associatedwith a conversion action representing the desired behavior of a userafter being presented with the content. Thus, the conversion action isan objective of the content, and may be the user clicking on orinteracting with the content or may be a user installing a program orapplication related to the content, exploring products available forpurchase and so forth. To evaluate the efficacy of the content slot, theconversion rate at the slot and in the benchmark for each selectedcontent item is identified to determine the ratio of conversions at thepublisher slot and in the benchmark. Thus a given content item whenpresented in the slot may have a conversion ratio of 110% or 90% of theconversion rate when presented in the benchmark, for example. Acrossdifferent content items, these conversion ratios may vary. In addition,some content may be more effective than other content at generatingrevenue or other desired value for the publisher when the content isselected for display. The conversion ratios may be weighed according tothe value to the publisher associated with each placed content item. Bycombining the conversion ratios for content items placed at the slot,the efficacy of displaying the content item to users at a given slot maybe determined. This efficacy may permit publishers to develop andcontrol slot placements to improve the efficacy of the slot relative tothe benchmark, and provide an evaluation of the slot that is independentof variations in third party value over time. In addition, because eachadvertisement may consider a conversion differently, the score mayaccount for different ways in which advertisers view success (i.e., thedifferent conversion events for sponsored content), which may differ.

The online system may also analyze each slot in a publisher's layout todetermine the efficacy of the slots within the publisher's layout. Forexample, a publisher may arrange a set of slots within an application.To determine the value of the conversion ratios for a given publisheracross a layout for the publisher as a whole, the online system usesinformation about online system users and their interactions with thirdparty content featured within the different locations of slots withinthe publisher's page layout. By evaluating the slots with respect toperformance on the benchmark placements and for the content items placedat the slots, the arrangement of slots in relation to one another can bemanipulated by the publisher to yield optimal conversion rates forcontent slots. For example, the publisher may re-position slots within agiven page, remove underperforming slots, or relocate slots to otherpages, to increase the efficacy of slots at the publisher and avoidslots that diminish user interactions.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a system environment for evaluating contentplacement options against benchmark placements according to oneembodiment.

FIG. 2 is a block diagram of an online system having a performanceoutcome score generator according to one embodiment.

FIG. 3A is a block diagram of an example content publisher according toone embodiment.

FIG. 3B is a block diagram of a benchmark publisher according to oneembodiment.

FIG. 4 is a table illustrating an example performance outcome scoreaccording to one embodiment.

FIG. 5A is a block diagram illustrating the page layout flow ofsponsored content slots within an example content publisher according toone embodiment.

FIG. 5B is a block diagram illustrating an alternate page layout flow ofsponsored content slots within an example content publisher according toone embodiment.

FIG. 6 is a flowchart illustrating a process for generating aperformance outcome score for a given sponsored content slot accordingto one embodiment.

FIG. 7 is a flowchart illustrating a process for providing sponsoredcontent to users of a content publisher according to one embodiment.

The figures depict various embodiments of the present invention forpurposes of illustration only. One skilled in the art will readilyrecognize from the following discussion that alternative embodiments ofthe structures and methods illustrated herein may be employed withoutdeparting from the principles of the invention described herein.

DETAILED DESCRIPTION I. System Overview

FIG. 1 is a block diagram of a system environment for generating andproviding performance outcome scores to content publishers. The systemenvironment includes an online system 100, a client device 110, anetwork 120, one or more third party systems 130, and a contentpublisher 140. In other embodiments, different and/or additionalentities can be included in the system architecture. The online system100 provides content items, such as sponsored content items, to users ofthe client devices 110 accessing the content publisher 140 or anapplication controlled by the content publisher 140. Sponsored contentitems are also provided to users by other publishing platforms, one ofwhich may be considered a benchmark for the sponsored content item. Theonline system 100 evaluates performance of the content items in slots ofthe content publisher 140 in comparison to the benchmark publishingplatform to determine the comparative performance of the contentpublisher's slots in promoting the content items and associatedinteractions with the content items. As the evaluation may be performedwith respect to individual slots in which content items can be provided,the content publisher 140 may use the performance of slots to adjustwhere and when slots are provided by the publisher to increase thecomparative performance of a publisher's slots. In addition, thecomparative performance of a publisher's slot may also reflect poorpresentation of content in the slot, and the online system 100 may alsouse the comparative scores to modify what content items may be presentedin the publisher's slot.

The online system 100 includes a computing environment that allows usersof the online system 100 to communicate or otherwise interact with eachother and access content. The online system 100 stores information aboutthe users, for example, user profile information and information aboutactions performed by users on the online system 100. The online system100 maintains content items for presentation to users via publicationplatforms, such as the content publisher 140 or via another publicationchannel, which may be operated by the online system 100. When anavailable slot is identified, the online system selects a content itemfor presentation in the available slot from various content items thatmay be presented to users. As users interact with the content publisher140 (or an application provided by the content publisher), the onlinesystem 100 provides content items for available slots. The selectedcontent items may be provided by third party systems 130 and may includea value to a third party system 130 when an action is performed by auser responsive to the selection of the content item. The online system100 may select from among these content items based on the valueassociated with each content item.

The client device 110 is a computing device capable of receiving userinput as well as transmitting and/or receiving data via the network 120.In one embodiment, a client device 110 is a conventional computersystem, such as a desktop or laptop computer. Alternatively, a clientdevice 110 may be a device having computer functionality, such as apersonal digital assistant (PDA), a mobile telephone, a smartphone oranother suitable device. A client device 110 is configured tocommunicate via the network 120. In one embodiment, a client device 110executes an application allowing a user of the client device 110 tointeract with the online system 100. For example, a client device 110executes a browser application to enable interaction between the clientdevice 110 and the online system 100 via the network 120. In anotherembodiment, a client device 110 interacts with the online system 100through an application programming interface (API) running on a nativeoperating system of the client device 110, such as IOS® or ANDROID™. Theclient device 110 may interact with the content publisher 140 to viewand retrieve content from the content publisher 140. In someembodiments, the client device 110 retrieves and executes an applicationprovided by the content publisher 140. Within the application (or whilethe client device 110 accesses content at the third party system 130),content slots may be specified for which content is retrieved from theonline system 100. The content slots may be located within the contentof the content publisher 140 according to a layout specified by thecontent publisher 140. When content items are retrieved from the onlinesystem 100, users may interact with the retrieved content items, forexample to access a related page or perform another interactionassociated with the selected content item.

One or more third party systems 130 may be coupled to the network 120for communicating with the online system 100. The third party systems130 provide content items to the online system 100 for selection andpresentation to users of the client devices 110. The content items maydescribe applications for execution by a client device 110 or othercontent for a user to interact with. The third party system 130 mayprovide the content items as sponsored content that encourages a user'saction based on placement of the content item to a user, and may alsoprovide a value to the online system and/or content publisher 140 whenthe content item is provided to the user or when the user performed thedesignated action. For example, the third party system 130 may provideinformation about products or services provided by the third partysystem that may be of interest to users. When users express an interestin or interact with the content, the third party system 130 may providevalue to the online system 100 or the content publisher 140 forproviding the content to users. Thus, the third party system 130 mayrepresent a company offering a product, service, or message that thecompany wishes to promote to users of the client devices 110.

The network 120 includes any combination of local area and/or wide areanetworks, using both wired and/or wireless communication systems. In oneembodiment, the network 120 uses standard communications technologiesand/or protocols. For example, the network 120 includes communicationlinks using technologies such as Ethernet, 802.11, worldwideinteroperability for microwave access (WiMAX), 3G, 4G, code divisionmultiple access (CDMA), digital subscriber line (DSL), etc. Examples ofnetworking protocols used for communicating via the network 180 includemultiprotocol label switching (MPLS), transmission controlprotocol/Internet protocol (TCP/IP), hypertext transport protocol(HTTP), simple mail transfer protocol (SMTP), and file transfer protocol(FTP). Data exchanged over the network 120 may be represented using anysuitable format, such as hypertext markup language (HTML) or extensiblemarkup language (XML). In some embodiments, all or some of thecommunication links of the network 120 may be encrypted using anysuitable technique or techniques.

The content publisher 140 provides content accessed by users of theclient device 110 and provides slots in which content items selected bythe online system 100 are placed. In one embodiment, a content publisher140 is an application provider that provides an application forexecution by a client device 110 or communicating data to client devices110 for use by an application executing on the client device 110. Inother embodiments, a content publisher 140 provides content (e.g.,videos, pictures, news stories, and other content) or other informationfor presentation via a client device 110.

The content publisher 140 features locations within a page layout whereone or more content items can be presented to a user from the onlinesystem 100. These content items may be sponsored by the third partysystems 130 to provide value to the content publisher 140. Sponsoredcontent items can be placed within sponsored content slots arrangedvertically or horizontally in different portions of a page layoutaccording to the preferences of a content publisher 140. In oneembodiment, locations within a page layout designated for sponsoredcontent are segmented into a number of sponsored content slots arrangedaccording to the dimensions of the sponsored content provided by a thirdparty system 130. This sponsored content is selected by the onlinesystem 100 for presentation on a client device 110 at a contentpublisher 140's page or within a content publisher's applicationexecuting on client device 110. The online system 100 may evaluate theefficacy of content slot by determining a performance outcome score fora content slot.

II. Online System

FIG. 2 is a block diagram of an online system 100 with a performanceoutcome score generator 270 according to one embodiment. In theembodiment illustrated in FIG. 2, the online system 100 includes a userprofile store 200, an action logger 210, an action log 220, an edgestore 250, a content store 260, a sponsored content selection module280, and a performance outcome score generator 270. In otherembodiments, the online system 100 may include additional, fewer, ordifferent components for various applications. Conventional componentssuch as network interfaces, security functions, load balancers, failoverservers, management and network operations consoles, and the like arenot shown so as to not obscure the details of the system architecture.

Each user of the online system 100 is associated with a user profile,which is stored in the user profile store 200. A user profile includesdeclarative information about the user that was explicitly shared by theuser and may also include profile information inferred by the onlinesystem 100. In one embodiment, a user profile store 200 of an onlinesystem user includes multiple data fields, each describing one or moreattributes of the user. Examples of information stored in a user profilestore 200 include biographic, demographic, and other types ofdescriptive information, such as work experience, educational history,gender, hobbies or preferences, location and the like. A user profilemay also store other information provided by the user, for example,images or videos. In certain embodiments, an image of a user may betagged with information identifying the online system user displayed inan image. A user profile in the user profile store 200 may also maintainreferences to actions by the corresponding user performed on contentitems in the action log 220.

While user profiles in the user profile store 200 are frequentlyassociated with individuals, allowing individuals to interact with eachother via the online system 100, user profiles may also be stored forentities such as businesses or organizations. This allows an entity toestablish a presence on the online system 100 for connecting andexchanging content with other online system 100 users. The entity maypost information about itself, about its products or provide otherinformation to users of the online system 100 using a brand pageassociated with the entity's user profile. Other users of the onlinesystem 100 may connect to the brand page to receive information postedto the brand page or to receive information from the brand page. A userprofile associated with the brand page may include information about theentity itself, providing users with background or informational dataabout the entity.

The action logger 210 receives communications about user actionsinternal to and/or external to the online system 100, populating theaction log 220 with information about user actions. Examples of actionsinclude adding a connection to another user, sending a message toanother user, uploading an image, reading a message from another user,viewing content associated with another user, and attending an eventposted by another user. In addition, a number of actions may involve anobject and one or more particular users, so these actions are associatedwith those users as well and stored in the action log 220.

The action log 220 may be used by the online system 100 to track useractions on the online system 100, as well as actions on third partysystems 130 that communicate information to the online system 100. Usersmay interact with various objects on the online system 100, andinformation describing these interactions is stored in the action log220. Examples of interactions with objects include: viewing videos,commenting on posts, sharing links, checking-in to physical locationsvia a mobile device, accessing content items, and any other suitableinteractions. Additional examples of interactions with objects on theonline system 100 that are included in the action log 220 include:viewing videos posted by a user's connections in the online system 100,commenting on a photo album, communicating with a user, establishing aconnection with an object, joining an event, joining a group, creatingan event, authorizing an application, using an application, expressing apreference for an object (“liking” the object), and engaging in atransaction. Additionally, the action log 220 may record a user'sinteractions with sponsored content on the online system 100 as well aswith other applications operating on the online system 100. In someembodiments, data from the action log 220 is used to infer interests orpreferences of a user, augmenting the interests included in the user'suser profile store 200 and allowing a more complete understanding ofuser preferences.

In one embodiment, the edge store 250 stores information describingconnections between users and other objects on the online system 100 asedges. Some edges may be defined by users, allowing users to specifytheir relationships with other users. For example, users may generateedges with other users that parallel the users' real-life relationships,such as friends, co-workers, partners, and so forth. In one embodiment,the user profile store 200 stores data describing the connectionsbetween different users of the online system 100, such as the number offriends shared between the users out of the total number of friends, thefraction of time since joining or becoming a member of the socialnetworking system that overlaps between the two users (e.g., whether theusers joined the online system at the same time or have an overlap for acertain period of time), or a combination of these signals. The recordof users and their connections in the online system 100 may be called a“social graph.”

Other edges are generated when users interact with objects in the onlinesystem 100, such as expressing interest in a page on the online system100, sharing a link with other users of the online system 100, viewingvideos posted by other users of the online system 100, and commenting onposts or videos provided by other users of the online system 100. Theconnections between users and other objects, or edges, can beunidirectional (e.g., a user following another user) or bidirectional(e.g., a user is a friend with another user).

In one embodiment, an edge may include various features eachrepresenting characteristics of interactions between users, interactionsbetween users and objects, or interactions between objects. For example,features included in an edge describe rate of interaction between twousers, how recently two users have interacted with each other, the rateor amount of information retrieved by one user about an object, or thenumber and types of comments posted by a user about an object. Thefeatures may also represent information describing a particular objector user. For example, a feature may represent the level of interest thata user has in a particular topic, the rate at which the user logs intothe online system 100, or information describing demographic informationabout a user. Each feature may be associated with a source object oruser, a target object or user, and a feature value. A feature may bespecified as an expression based on values describing the source objector user, the target object or user, or interactions between the sourceobject or user and target object or user. Hence, an edge may berepresented as one or more feature expressions.

The edge store 250 also stores information about edges, such as affinityscores for objects, interests, and other users. In one embodiment,affinity scores, or “affinities,” are computed by the online system 100over time to approximate a user's interest in an object or another userin the online system 100 based on the actions performed by the user. Auser's affinity may be computed by the online system 100 over time toapproximate a user's affinity for an object, interest, and other usersin the online system 100 based on the actions performed by the user.Computation of affinity is further described in U.S. patent applicationSer. No. 12/978,265, filed on Dec. 23, 2010, U.S. patent applicationSer. No. 13/690,254, filed on Nov. 30, 2012, U.S. patent applicationSer. No. 13/689,969, filed on Nov. 30, 2012, and U.S. patent applicationSer. No. 13/690,088, filed on Nov. 30, 2012, each of which is herebyincorporated by reference in its entirety. Multiple interactions betweena user and a specific object may be stored as a single edge in the edgestore 250, in one embodiment. Alternatively, each interaction between auser and a specific object is stored as a separate edge.

The content store 260 stores objects that represent various types ofcontent. Examples of content represented by an object include a video,page post, status update, photograph, link, shared content item, gamingapplication achievement, check-in event at a local business, brand page,or any other type of content. Online system 100 users may create objectsstored by the content store 260, such as status updates, photos taggedby users to be associated with other objects in the online system 100,events, groups, or applications. In some embodiments, objects arereceived from third party systems 130 and placed in the content store260. These objects may represent sponsored content campaigns provided byone or more third party systems 130. In one embodiment, receivedinformation associated with sponsored content campaigns includescriteria regarding how and when the sponsored content is to be used bythe online system 100, which may include a threshold performance metricfor sponsored content slots within a content publisher 140. Thus, thesponsored content may specify a minimum performance outcome score for aslot in which the sponsored content is presented. As discussed below,the performance outcome score may specify the comparative likelihood ofusers to interact with a sponsored content slot relative to a benchmarkslot. The performance outcome score may also reflect user experiences atthe different placement locations. By setting a threshold performancescore, the third party may limit presentation of the advertisement toavoid slots that present poor user experiences.

The sponsored content may also specify information about selecting thesponsored content, such as an amount for the sponsored content to bid onplacements of the sponsored content in an auction, or a budget for acampaign associated with the sponsored content. The sponsored contentmay also specify parameters or conditions for providing the content tousers. The sponsored content may also specify one or more actionsperformed by a user that is desired by the third party provider for thesponsored content. In this way, the desired action is a measurableobjective of the sponsored content. The desired action is termed aconversion or a conversion event. The conversion event may be the userinteracting with the content, for example by “clicking” or selecting thecontent to view a page or other item referenced by the sponsoredcontent. In other examples the conversion event may be downloading orinstalling an application associated with the sponsored content, whileanother conversion event may be installing the application and furtherpurchasing or interacting with the application. In other examples, theconversion event may be completing a purchase on a website. Theseconversion events may be specified for each sponsored content by thethird party that provided the sponsored content item. In some examples,the online system receives a value or contribution amount from the thirdparty for placing a sponsored content item when the conversion eventoccurs.

In other examples, a conversion event for the sponsored content item maybe automatically determined based on the type of sponsored content,which may not be the basis on which the online system receives value forpresenting the sponsored content. In this example, the online system 100may identify or determine an objective (e.g., conversion event) for anadvertisement to reflect the likely desired objective of theadvertisement. For example, a sponsored content item relating to aninstallation of an application may be associated with a conversion eventfor installing the application. As another example, for a sponsoredcontent item relating to an installation of an application that reportsapplication events to the online system, the online system 100 mayidentify a conversion event related to receipt of specified events fromthe application indicating that a user both installed and interactedwith the application. These interactions may include a validation of theapplication, a purchase event in the application, or an event in theapplication indicating interaction with the application beyondinstalling the application. The conversion events for a sponsoredcontent item may be stored and identified in the action log 220 or theedge store 250.

The sponsored content selection module 280 selects sponsored contentitems to be presented to an online system 100 user. The selected contentmay be provided to the user when the user accesses the online system100, for example to view content of the online system 100, or may beprovided to the user when the user accesses the content publisher 140 orinteracts with a content publisher's application. When the online system100 or a content publisher 140 (or the content publisher's applicationon the device) has a location (or slot) in which sponsored content maybe placed by the online system 100, the sponsored content selectionmodule 280 receives a request to select content for the location andprovides the selected content item for display in that location. In oneembodiment, the sponsored content selection module 280 selects sponsoredcontent items from the available sponsored content campaigns stored inthe content store 260. In other embodiments, the sponsored contentselection module 280 selects sponsored content item from availablesponsored content campaigns stored in another system, external to theonline system 100. The sponsored content selection module 280 examinescriteria associated with each sponsored content campaign (e.g.,targeting criteria) and selects one or more sponsored content items forpresentation to the user. To select the sponsored content, as oneexample, the sponsored content selection module 280 identifies sponsoredcontent that target a particular user, and perform an auction forplacement in the slot based on the expected value of each sponsoredcontent item for placement in the slot.

The performance outcome score generator 270 determines the efficacy ofsponsored content slots containing sponsored content presented by acontent publisher 140 compared to those placed within the online system100 or another location that serves as a benchmark placement. Forexample, the online system 100 may present sponsored content in portionsof a newsfeed or a portion of the page on the online system 100. Toevaluate the sponsored content slot of a publisher, the performanceoutcome score generator 270 identifies sponsored content that wasselected for presentation at both the benchmark (e.g., the online system100) and the a content publisher 140 page and the online system 100. Theperformance outcome score generator 270 determines conversion ratemetrics associated with both locations and generates a performanceoutcome score. This performance outcome score can be used by contentpublishers 140 to determine the optimal placement of individualsponsored content slots within the layout of a page as well as theoverall layout of sponsored content slots in relation to one another ina series of pages. The layout of sponsored content slots may also betermed “page layout flow.”.

The performance outcome score generator 270 generates a performanceoutcome score used to define the efficacy of sponsored content slotswithin a content publisher 140. This efficacy is gauged by sponsoredcontent conversion rate metrics that contribute to the overallperformance outcome score for a given sponsored content slot. Eachsponsored content slot may receive its own performance outcome scorebased on a benchmark placement within the online system 100

The performance outcome score generation is based on a comparisonbetween conversion rate metrics yielded from sponsored content slotsfeatured in the online system 100 against those produced on a contentpublisher 140. FIG. 3A provides an example of a sponsored content slot300 featured on a content publisher 140. FIG. 3B illustrates an exampleplacement of sponsored content slots 300 within the news feed 310 andnews ticker 320 presented by the online system 100. To compareconversion rate metrics generated by the sponsored content slot 300featured on the content publisher 140 against those generated by theslots in the online system 100, sponsored content placed in bothlocations is kept identical (i.e., same sponsored content campaign).Responsive to loading a content publisher 140 page containing sponsoredcontent slots, the content publisher 140 (or the publisher's applicationon the client device) sends a sponsored content request to the onlinesystem 100 to receive sponsored content to place within the sponsoredcontent slot 300 for presentation to the user. As discussed above, thesponsored content selection module 280 examines available contentcampaigns, and makes a selection based on criteria associated with eachcampaign.

As discussed above, each sponsored content campaign affiliated with athird party system 130 may contain criteria regarding when and how topresent the sponsored content to users. In one embodiment, criteriaaffiliated with sponsored content campaigns are stored with thesponsored content in the content store 260 in the online system 100. Inother embodiments, these criteria may be stored in a separate system tobe used by the online system 100 when needed. Sponsored content campaigncriteria may specify: a third party contribution amount (e.g., a valueassociated with placing the content or a conversion event), targetingcriteria, sponsored content type, the objective of the sponsored content(i.e., a conversion event), sponsored content slot type, and a thresholdperformance outcome score (i.e., threshold publisher conversion rate).

The third party contribution amount is associated with sponsored contentprovided by a third party system 130 and specifies an amount ofcompensation the third party system 130 provides the online system 100if the sponsored content is presented to a user or accessed by a user.The contribution amount may alternatively (or in addition) specify anamount when a user performs a conversion event. In one embodiment, thethird party contribution amount is used by the online system 100 todetermine an expected value, such as monetary compensation, received bythe online system 100 for presenting the content to a user, if thecontent receives a user interaction, or based on any other suitablecondition. For example, the third party contribution amount specifies amonetary amount that the online system 100 receives from the third partysystem 130 if the conversion event occurs and the expected value isdetermined based on the third party contribution amount and aprobability of a user performing the conversion event.

Targeting criteria describe a target group comprised of specific usersof online system 100 users. Targeting is generalized based on usercharacteristics as determined by the characteristics of a subset ofusers of the online system 100. Characteristics may include demographicinformation about users of the online system 100, such as age, gender,political views, education status, college year, relationship status,gender(s) interested in dating, and geographic region information. Thus,the target group may be identified in advance as a list of users, or thetarget group may be identified as a statement of characteristics ortargeting criteria for those users, and the sponsored content selectionmodule 280 may determine at impression time, when an opportunity toprovide an impression to a user presents itself, whether that user fallsin the target group or not. In other words, the target group may be alist of identifiers in the online system 100 of particular users whohave been determined in advance to meet the targeting criteria, or thetarget group may simply be identified as the targeting criteria itselfand the users falling within the targeting criteria may be identified atimpression time.

Sponsored content type indicates to the sponsored content selectionmodule 280 which category of sponsored content is to be presented in asponsored content campaign. The sponsored content type may be used todetermine a conversion event or objective for the sponsored content.Examples of sponsored content types include mobile app install ads(MAIA) with purchase events, MAIA with activation events only, MAIA withno events, mobile app engagement, and website conversion. Examples ofobjectives associated with these sponsored content types includepurchase events, activations, installs, app events, and deep funnelconversion, respectively. As discussed above, these conversion eventsmay be identified for each sponsored content, even when the contributionamount is not contingent on the conversion event. Objectives arefulfilled when an online system 100 user completes the action specifiedby the sponsored content type.

Sponsored content may also specify criteria for types of slots in whichsponsored content is to be placed. The sponsored content slots may be ofdifferent dimensions and vary proportions, each of which may be suitablefor different types of sponsored content. For example, a landing page ofa content publisher 140 may specify a banner-size sponsored content slot300 as shown in FIG. 3A. Additional examples of sponsored content slottypes and their respective placements include: banner slots presentingsponsored content on the top or bottom of a page or screen; interstitialslots presenting sponsored content between portions of publishercontent, such as on a loading page or a popup window; and sidebar slotspresenting sponsored content on the side of a display. As discussedherein, these slots may be evaluated to identify an advertiser outcomescore for the slot.

A sponsored content campaign may also specify a threshold performanceoutcome score that limits presentation of the sponsored content to witha baseline efficacy for sponsored content slot placement on which thethird party system 130 is willing to place a third party contribution.For example, if a content publisher 140 page features sponsored contentslots that exhibit sufficiently lower conversion rate metrics than thosein a benchmark system (e.g., the online system 100) while displayingsimilar (or the same) sponsored content, a third party system 130 maynot wish to place a third party contribution for presentation on thatparticular page. Conversely, if a content publisher 140 page producesgreater conversion rate metrics than those generated by the onlinesystem 100, the third party system 130 may decide to place a third partycontribution on the content publisher 140 sponsored content slots forthe presentation of their sponsored content. In this way, the conversionrate metrics generated by the sponsored content slots within the onlinesystem 100 serve as a benchmark for the conversion rate metrics producedby content publishers 140 external to the online system 100 and may beused to control which slots are eligible to receive certain sponsoredcontent.

The performance outcome score generator 270 can use information from theonline system 100 to generate sponsored content conversion rate metricsaffiliated with individual sponsored content slots displayed on acontent publisher 140. In one embodiment, the performance outcome scoregenerator 270 can use information about an online system 100 usercollected by the action logger 210, stored in the action log 220, toidentify actions taken by the online system 100 user while (and after)interacting with third party sponsored content. For example, a contentpublisher 140 may recognize a user of an online system 100 through asocial plug-in enabling the content publisher 140 to identify the userof the online system 100. Content publishers 140 and other systemsaccessed by the client device 110 may communicate information about auser's actions outside of the online system 100 to the online system 100for association with the user. Hence, the action log 220 may recordinformation about actions users perform on other systems to determinewhen a user performs a conversion event and associate the conversionevent with the user who viewed sponsored content on the client device.

FIG. 4 provides an example use case in which the performance outcomescore generator 270 generates a performance outcome score for asponsored content slot 450 featured on a content publisher 140. In thisexample, the content publisher provides an application executing on theclient device that retrieves and presents sponsored content to the useras the user interacts with the application, for example in a “brickgame.” To determine the performance outcome score, sponsored contentcampaigns are evaluated with respect to the conversion rates for thatsponsored content campaign at the sponsored content slot 450 and at thesponsored content slot 460 of the benchmark publisher 310. The sponsoredcontent presented at both the content publisher 140 and the benchmarkpublisher 310 may be evaluated to identify a consistent set of sponsoredcontent presented. In this example, three items of sponsored content areillustrated as being presented in both locations, although many more maybe evaluated in practice.

To evaluate the sponsored content slot 450 with respect to eachpublisher, the conversion rate is determined for each item of sponsoredcontent at the sponsored content slot 450 and at the benchmark sponsoredcontent slot 460. Thus, a publisher conversion rate 400 is determined bythe performance outcome score generator 270 for each of the threesponsored content campaigns (sponsored content 1, 2, and 3) previouslyfeatured within the sponsored content slot 400 on the content publisher140. In addition, the performance outcome score generator 270 determinesa benchmark conversion rate 410 for each sponsored content presented insponsored content slot 460, for example located within the news feed 320or news ticker 330 of the benchmark publisher 310 for the same threesponsored content campaigns. The conversion rate may be determined bydetermining the percentage, or rate, of users presented with thesponsored content that later performed the objective or conversion eventassociated with the sponsored content. The objective or conversion eventmay be specified by the third party providing the sponsored content, ormay be determined based on a type of the sponsored content or otherevents tracked in relation to the sponsored content (e.g., activationevents of an application). Thus, each item of sponsored content may havean associated conversion event that may differ by event type with otherevaluated sponsored content. For example, the conversion rate forsponsored content 1 may be based on user interactions with an installedapplication, while the conversion rate for sponsored content 2 may bebased on a user purchase at a webpage.

Conversion rate metrics generated by the content publisher 140 and thosefrom the benchmark publisher 310 are used to determine a placementconversion ratio 420. For example, the performance outcome scoregenerator 270 uses publisher conversion rate 400 metrics for sponsoredcontent 1 (80%) against benchmark conversion rate 410 metrics for thesame sponsored content 1 (84%) to produce a placement conversion ratio420 (95.2%). This placement conversion ratio 420 is the quotientproduced by the publisher conversion rate 400 (dividend) and benchmarkconversion rate 410 (divisor):

${{placement}\mspace{14mu} {conversion}\mspace{14mu} {ratio}} = {\frac{{publisher}\mspace{14mu} {conversion}\mspace{14mu} {rate}}{{benchmark}\mspace{14mu} {conversion}\mspace{14mu} {rate}} = {\frac{80\%}{84\%} = {95.2\%}}}$

The performance outcome score generator 270 uses information from thecontent store 260 to determine a third party contribution 430 put forthby the third party system 130 that provided sponsored content 1. Thecontribution may reflect the total amount of contribution by the thirdparty of sponsored content 1 for presenting the sponsored content tousers. This contribution ($50), as well as the third party contribution430 values for sponsored content 2 and 3, is used to weight theplacement conversion ratio 420, producing a weighted conversion ratio440 (0.136) for sponsored content 1:

${{weighted}\mspace{14mu} {conversion}\mspace{14mu} {ratio}} = {\frac{95.2 \times {third}\mspace{14mu} {party}\mspace{14mu} {contribution}}{{total}\mspace{14mu} {third}\mspace{14mu} {party}\mspace{14mu} {contribution}} = {\frac{95.2 \times 50}{350} = 0.136}}$

The performance outcome score generator 270 combines weighted conversionratios 440 produced for all three sponsored content campaigns (0.136,0.295, and 0.457) to yield an average conversion ratio of 0.888. Thus,the placement conversion ratio may be weighed according to thecontribution amounts. This value is used to determine the performanceoutcome score for the sponsored content slot 300 featured on the contentpublisher 140:

average conversion ratio=0.136+0.295+0.457=0.888

In some examples, the performance outcome store is the averageconversion ratio of the sponsored content, while in other examples theperformance outcome store is further weighted based on the third partycontribution associated with each sponsored content. In additionalexamples, the performance outcome store is converted from the ratio to apositive numerical value that logarithmically increases. For example,one embodiment of the performance outcome store is:

performance outcome score=min(log₂(0.888)+7,1)=6.83

The performance outcome score may be used for a variety of purposes. Asdiscussed previously, the performance outcome score may be used by asponsored content provider as criteria for displaying sponsored content,such that sponsored content is not be presented if the performanceoutcome store does not exceed a certain threshold. In addition, thescore may be reported to the content publisher 140 for further analysisand testing (e.g., AB testing) to determine methods for improving theperformance outcome score of the content presented by the contentpublisher 140. As discussed below, the content publisher 140 may modifythe location, number, or sequence of sponsored content slots to increasethe performance outcome score. The performance outcome score may begenerated and provided to a content publisher 140 to identifycomparatively low-performing content slots, which the content publishermay use to remove sponsored content slots that do not result ineffective user interactions or lead to conversion events (compared tothe conversion event frequency of the same sponsored content at thebenchmark system).

Using the performance outcome scores, the publisher may evaluate slotswithin a layout of slots, or within a “flow” of the publisher'sapplication. Thus, the placements of slots relative to one anotherwithin the composition of one or more pages of a content publisher 140can be evaluated with the performance outcome score. This allows thecontent publisher 140 to compare separate sponsored content placementand page layout flows. For example, if the content publisher 140replaces an interstitial slot with a banner slot, the flow basedperformance outcome score may change to yield a higher publisherconversion rate 400.

FIG. 5A illustrates an example content publisher 140 page layout flow ofa game application, Brick Game. In this example, the landing page 500contains a banner slot 510, the loading page 520 features aninterstitial slot, the first level of game play displays a sidebar slot550, and the second level of game play contains an additional bannerslot 510. The banner slot 510 located on the landing page 500 is filledwith sponsored content provided by the online system 100 when an onlinesystem 100 user requests to load the page (e.g., open the gameapplication from a client device 110). The interstitial slot 530 isfilled when the game application advances from the landing page 500 tothe loading page 520 as it begins loading the next state of the gameapplication (e.g., loading level 1 for user game play). The sidebar slot550 is filled and presented to the online system 100 user during actuallevel 1 game play 540 within the right portion of the user's screen.Sponsored content that fills the sidebar slot 550 is presentedthroughout level 1 game play 540. The additional banner slot 510 isfilled with sponsored content and presented when the online system 100user advances to level 2 game play 560 within the top portion of theuser's screen. Similar to sponsored content presented in the sidebarslot 550, sponsored content that fills the banner slot 510 is presentedthroughout level 2 game play 560. Thus, these slots within theadvertisement may present a set of slots that may be evaluated with aperformance outcome score. Each slot illustrated in FIG. 5A is assigneda performance outcome score by the performance outcome score generator270 as described above. However, in order to identify how sponsoredcontent slots 300 perform in relation to one another (e.g., differentplacements within the page layout), the content publisher 140 may electto implement further testing and analysis to manipulate page layout flowto yield optimal publisher conversion rate 400 metrics.

FIG. 5B illustrates an example manipulation of page layout flow in whichthe same content publisher 140, Brick Game, has altered the sponsoredcontent slot types and placements level 1 game play 540 and level 2 gameplay 560 based on performance outcome scores. In relation to FIG. 5A,the sidebar slot 550 from level 1 game play 540 has been replaced with abanner slot 510 to be filled with sponsored content before the userinteracts with the level. In addition, the banner slot 510 featured inlevel 2 game play 560 in FIG. 5A has been replaced with a sidebar slot550 in FIG. 5B. Further page layout flow manipulation is shown in FIG.5B as the placement of the sidebar slot 550 relative to general gameplay has been moved from the right portion of the user's screen in FIG.5A to the left portion. These changes affect the way that sponsoredcontent is presented to an online system 100 user interacting with thecontent publisher 140, and, resultantly, affect flow based outcomescore. Thus, the performance outcome scores for the set of sponsoredcontent slots permits the application publisher to effectively evaluatealternative flows and slot placements within the application. Becausethe slots are compared to the benchmark and the comparison for a givenslot is based on the content presented in the publisher's slot and inthe benchmark placement, the effectiveness of each slot may be moredirectly compared relative to comparing the raw conversion rates of theslots against one another, in which case the content presented at eachslot may vary along with the particular content presented in each slot.

III. Performance Outcome Score Process

FIG. 6 is a flowchart illustrating a process for generating aperformance outcome score for a given sponsored content slot accordingto one embodiment. In the embodiment illustrated in FIG. 6, theperformance outcome score generator 270 identifies 600 a sponsoredcontent slot 300 displayed within a content publisher 140 forevaluation. The performance outcome score generator 270 identifies 610sponsored content campaigns that were selected by the sponsored contentselection module 280 for presentation within the sponsored content slot300 from those campaigns stored in the content store 260. Theperformance outcome score generator 270 monitors 620 benchmarkconversion rate 410 metrics generated by sponsored content slots 300placed in within a benchmark placement (e.g., the online system 100) forthe identified sponsored content campaigns. In addition, the performanceoutcome score generator 270 monitors 630 publisher conversion rate 400metrics produced by sponsored content slots 300 placed within one ormore pages of a content publisher 140. The performance outcome scoregenerator 270 determines 640 the third party contribution 430 amountassociated with each of the sponsored content campaigns selected forpresentation within the sponsored content slot 300. It also determines650 placement conversion ratios 420 for each of the selected sponsoredcontent campaigns that compare publisher conversion rate 400 metrics tobenchmark conversion rate 410 metrics. The performance outcome scoregenerator 270 combines 660 placement conversion ratios 420, eachweighted by its respective third party contribution 430, to produce anoverall performance outcome score for the identified sponsored contentslot. This performance outcome score is reported 670 to the contentpublisher 140 for changes in page layout flow as needed.

FIG. 7 is a flowchart illustrating a process for providing sponsoredcontent to users of a content publisher 140 according to one embodiment.The content publisher 140 loads 700 a user requested page containingsponsored content slots 300 and transmits 710 a request to the sponsoredcontent selection module 280 for sponsored content from the onlinesystem 100 to place within specified sponsored content slots 300 forpresentation to the user. The content publisher 140 provides 720 theuser with sponsored content placed within the specified sponsoredcontent slots 300, and receives 730 performance outcome scores for thosesponsored content slots 300 from the online system 100. It uses theperformance outcome scores to test 740 page layout to determine the mostefficient sponsored content slot 300 arrangement to garner the greatestpublisher conversion rate 400 metrics. The content publisher 140 canthen alter 750 page layout to accommodate an optimal page layout flowfor maximizing publisher conversion rate 400 metrics.

SUMMARY

The foregoing description of the embodiments of the invention has beenpresented for the purpose of illustration; it is not intended to beexhaustive or to limit the invention to the precise forms disclosed.Persons skilled in the relevant art can appreciate that manymodifications and variations are possible in light of the abovedisclosure.

Some portions of this description describe the embodiments of theinvention in terms of algorithms and symbolic representations ofoperations on information. These algorithmic descriptions andrepresentations are commonly used by those skilled in the dataprocessing arts to convey the substance of their work effectively toothers skilled in the art. These operations, while describedfunctionally, computationally, or logically, are understood to beimplemented by computer programs or equivalent electrical circuits,microcode, or the like. Furthermore, it has also proven convenient attimes, to refer to these arrangements of operations as modules, withoutloss of generality. The described operations and their associatedmodules may be embodied in software, firmware, hardware, or anycombinations thereof.

Any of the steps, operations, or processes described herein may beperformed or implemented with one or more hardware or software modules,alone or in combination with other devices. In one embodiment, asoftware module is implemented with a computer program productcomprising a computer-readable medium containing computer program code,which can be executed by a computer processor for performing any or allof the steps, operations, or processes described.

Embodiments of the invention may also relate to an apparatus forperforming the operations herein. This apparatus may be speciallyconstructed for the required purposes, and/or it may comprise ageneral-purpose computing device selectively activated or reconfiguredby a computer program stored in the computer. Such a computer programmay be stored in a non-transitory, tangible computer readable storagemedium, or any type of media suitable for storing electronicinstructions, which may be coupled to a computer system bus.Furthermore, any computing systems referred to in the specification mayinclude a single processor or may be architectures employing multipleprocessor designs for increased computing capability.

Embodiments of the invention may also relate to a product that isproduced by a computing process described herein. Such a product maycomprise information resulting from a computing process, where theinformation is stored on a non-transitory, tangible computer readablestorage medium and may include any embodiment of a computer programproduct or other data combination described herein.

Finally, the language used in the specification has been principallyselected for readability and instructional purposes, and it may not havebeen selected to delineate or circumscribe the inventive subject matter.It is therefore intended that the scope of the invention be limited notby this detailed description, but rather by any claims that issue on anapplication based hereon. Accordingly, the disclosure of the embodimentsof the invention is intended to be illustrative, but not limiting, ofthe scope of the invention, which is set forth in the following claims.

What is claimed is:
 1. A computer-implemented method comprising:identifying a sponsored content slot within a content publisher forevaluation of relative efficacy of the sponsored content slot relativeto a benchmark placement, wherein an online system selects content itemsfor placement in the sponsored content slot responsive to individualrequests for content from client devices interacting with the contentpublisher; identifying a plurality of sponsored content items previouslyselected for placement within the sponsored content slot; for each ofthe plurality of sponsored content items, determining a conversion ratioof a placement rate of desired user interaction with the sponsoredcontent item when the content item is placed at the sponsored contentslot compared to a benchmark rate of desired user interaction with thesponsored content item when the content item is placed at a benchmarkplacement; determining a performance outcome score associated with thesponsored content slot by combining the plurality of conversion ratios;and sending the performance outcome score associated with the sponsoredcontent slot to the content publisher.
 2. The computer-implementedmethod of claim 1, further comprising: determining a third-partycontribution associated with each of the plurality of sponsored contentitems, the third party contribution representing a value provided by athird party for placing the sponsored content in the sponsored contentslot; and wherein combining the plurality of conversion ratios todetermine the performance outcome score includes weighing the conversionratios according to the respective value provided by the third party. 3.The computer-implemented method of claim 1, wherein the desired useraction for a first sponsored content item differs from the desired useraction for a second sponsored content item.
 4. The computer-implementedmethod of claim 1, wherein the conversion ratio for a sponsored contentitem is determined for a look-back period of placing the sponsoredcontent items.
 5. The computer-implemented method of claim 4, whereinthe look-back period for a sponsored content item based on a volume ofplacements of the sponsored content item.
 6. The computer-implementedmethod of claim 4, wherein the look-back period for a first sponsoredcontent item differs from the look-back period for a second sponsoredcontent item.
 7. The computer-implemented method of claim 1, wherein thebenchmark placement corresponds to placement within sponsored contentslots within another online system.
 8. The computer-implemented methodof claim 1, wherein the benchmark placement is a previous placementlocation of the sponsored content slot within the content publisher. 9.The computer-implemented method of claim 1, further comprising:receiving a request for a content item for placement in the sponsoredcontent slot for display to a user; and identifying a set of sponsoredcontent items eligible for display to the user; identifying a subset ofsponsored content items from the set of sponsored content items based onthe performance outcome score of the sponsored content slot; selecting asponsored content item for placement in the sponsored content slot fromthe subset of sponsored content items; and sending the selectedsponsored content item for display to the user in the sponsored contentslot.
 10. The computer-implemented method claim 1, wherein the sponsoredcontent slot is one of a plurality of sponsored content slots in a firstset of sponsored content slots presented in a first application flow ofan application of a client device, the method further comprising:determining a first application performance outcome score for the firstset of sponsored content slots, based in part on the performance outcomescore for the sponsored content slot; determining a second applicationperformance outcome score for a second set of sponsored content slotsfor a second application flow of the application; and determining apreferred application flow from the first application flow and thesecond application flow based on the first application performanceoutcome score and the second application performance outcome score. 11.A non-transitory computer readable storage medium having instructionsencoded thereon that, when executed by a processor, cause the processorto perform the steps including: identifying a sponsored content slotwithin a content publisher for evaluation of relative efficacy of thesponsored content slot relative to a benchmark placement, wherein anonline system selects content items for placement in the sponsoredcontent slot responsive to individual requests for content from clientdevices interacting with the content publisher; identifying a pluralityof sponsored content items previously selected for placement within thesponsored content slot; for each of the plurality of sponsored contentitems, determining a conversion ratio of a placement rate of desireduser interaction with the sponsored content item when the content itemis placed at the sponsored content slot compared to a benchmark rate ofdesired user interaction with the sponsored content item when thecontent item is placed at a benchmark placement; determining aperformance outcome score associated with the sponsored content slot bycombining the plurality of conversion ratios; and sending theperformance outcome score associated with the sponsored content slot tothe content publisher.
 12. The non-transitory computer readable storagemedium of claim 1, further comprising: determining a third-partycontribution associated with each of the plurality of sponsored contentitems, the third party contribution representing a value provided by athird party for placing the sponsored content in the sponsored contentslot; and wherein combining the plurality of conversion ratios todetermine the performance outcome score includes weighing the conversionratios according to the respective value provided by the third party.13. The non-transitory computer readable storage medium of claim 1,wherein the desired user action for a first sponsored content itemdiffers from the desired user action for a second sponsored contentitem.
 14. The non-transitory computer readable storage medium of claim1, wherein the conversion ratio for a sponsored content item isdetermined for a look-back period of placing the sponsored contentitems.
 15. The non-transitory computer readable storage medium of claim4, wherein the look-back period for a sponsored content item based on avolume of placements of the sponsored content item.
 16. Thenon-transitory computer readable storage medium of claim 4, wherein thelook-back period for a first sponsored content item differs from thelook-back period for a second sponsored content item.
 17. Thenon-transitory computer readable storage medium of claim 1, wherein thebenchmark placement corresponds to placement within sponsored contentslots within another online system.
 18. The non-transitory computerreadable storage medium of claim 1, wherein the benchmark placement is aprevious placement location of the sponsored content slot within thecontent publisher.
 19. The non-transitory computer readable storagemedium of claim 1, further comprising: receiving a request for a contentitem for placement in the sponsored content slot for display to a user;and identifying a set of sponsored content items eligible for display tothe user; identifying a subset of sponsored content items from the setof sponsored content items based on the performance outcome score of thesponsored content slot; selecting a sponsored content item for placementin the sponsored content slot from the subset of sponsored contentitems; and sending the selected sponsored content item for display tothe user in the sponsored content slot.
 20. The non-transitory computerreadable storage medium claim 1, wherein the sponsored content slot isone of a plurality of sponsored content slots in a first set ofsponsored content slots presented in a first application flow of anapplication of a client device, the method further comprising:determining a first application performance outcome score for the firstset of sponsored content slots, based in part on the performance outcomescore for the sponsored content slot; determining a second applicationperformance outcome score for a second set of sponsored content slotsfor a second application flow of the application; and determining apreferred application flow from the first application flow and thesecond application flow based on the first application performanceoutcome score and the second application performance outcome score.